Wearable device including eye tracking apparatus and operation method of the wearable device

ABSTRACT

An eye tracking apparatus includes: a light source module including a light source and a scanning mirror, the light source being configured to provide light toward an eye region of a user while a direction of the light is changed at intervals of a predetermined time period under control of the scanning mirror; and a dynamic vision sensor (DVS) camera module configured to generate an image based on a variation in an amount of light reflected from the eye region of the user.

CROSS-REFERENCE TO RELATED APPLICATION

This application is based on and claims priority under 35 U.S.C. § 119to Korean Patent Application No. 10-2019-0171916, filed on Dec. 20,2019, in the Korean Intellectual Property Office, the disclosure ofwhich is incorporated by reference herein in its entirety.

BACKGROUND 1. Field

The disclosure relates to eye tracking apparatuses, wearable devicesincluding an eye tracking apparatus, operation methods of the eyetracking apparatus, and operation methods of the wearable devicesincluding the eye tracking apparatus, and more particularly, to an eyetracking apparatus for tracking a user's view by using a dynamic visionsensor (DVS) image, an operation method thereof, a wearable deviceincluding the eye tracking apparatus, and an operation method of thewearable device including the eye tracking apparatus.

2. Description of the Related Art

With the recent development of technologies, wearable devices that arewearable on human bodies have been provided in various forms. Amongthem, a glasses-type wearable device including a head mounted display isa wearable device to be worn on a user's head and is capable ofproviding an augmented reality (AR) service to the user by providingvisual information about a virtual object through a display.

When this wearable device is used, a user's view may be used for aninteraction between either a virtual object on the display or an actualobject present in a real-world space and the user. Accordingly, there isa demand for technologies for tracking a user's view by tracking auser's eye with low power and at high speed while being resilient tovariations in illumination.

SUMMARY

Provided are a wearable device capable pf tracking a user's view byusing a dynamic vision sensor (DVS) image, and an operation methodthereof.

Provided are a wearable device capable of detecting respective positionsof a cornea and a pupil of a user, and an operation method thereof.

Provided are non-transitory computer-readable recording medium havingrecorded thereon a computer program, which, when executed by at leastone processor, causes the at least one processor to perform the abovemethod(s). Technical problems to be solved are not limited to thosedescribed above and other technical problems may be present.

Additional aspects will be set forth in part in the description whichfollows and, in part, will be apparent from the description, or may belearned by practice of the presented embodiments of the disclosure.

In accordance with an aspect of the disclosure, there is provided an eyetracking apparatus, including: a light source module including a lightsource and a scanning mirror, the light source being configured toprovide light toward an eye region of a user while a direction of thelight is changed at intervals of a predetermined time period undercontrol of the scanning mirror; and a dynamic vision sensor (DVS) cameramodule configured to generate an image based on a variation in an amountof light reflected from the eye region of the user.

The scanning mirror may be further configured to change the direction ofthe light by controlling to change a direction in which the light isprojected from the light source toward the eye region along at least oneof a longitudinal straight line, a transverse straight line, or aLissajous curve.

In accordance with an aspect of the disclosure, there is provided awearable device including: a light source module including a lightsource and a scanning mirror, the light source configured to providelight toward an eye region of a user and the scanning mirror configuredto control to change a direction of the light from the light source atintervals of a predetermined time period; a dynamic vision sensor (DVS)camera module configured to generate an image based on a variation in anamount of light reflected from the eye region of the user; and aprocessor configured to detect a cornea region of an eye of the userfrom the image.

The processor may be further configured to detect a movement of thecornea region by using a plurality of images consecutively generated bythe DVS camera module based on variations in an amount of lightreflected from one or more points of the eye region.

The processor may be further configured to track a change in a positionof the detected cornea region based on the plurality of imagesconsecutively generated by the DVS camera module.

The processor may be further configured to detect a pupil region of theuser from the image, based on the detected cornea region.

The processor may be further configured to filter a peripheral regionoutside the cornea region from the image and detect the pupil regionbased on a result of filtering.

The processor may be further configured to: store a position of thedetected pupil region in a memory; and based on a change in the positionof the detected pupil region, update the position of the detected pupilregion and store the updated position in the memory.

The processor may be further configured to, based on the pupil regionnot being detected in the image, estimate a current position of thepupil region, based on the position of the pupil region stored in thememory.

The processor may be further configured to: determine a direction of aview of the eye based on a position of the cornea region and theposition of the pupil region, and store the positions of the cornearegion and the pupil region and the determined direction of the view ina memory.

The processor may be further configured to track a change in thedirection of the view based on a plurality of images consecutivelygenerated by the DVS camera module.

In accordance with an aspect of the disclosure, there is provided amethod of operating a wearable device, the method including: providinglight from a light source toward an eye region of a user while changinga direction of the light at intervals of a predetermined time period byusing a scanning mirror; and generating an image by using a dynamicvision sensor (DVS) camera module, based on a variation in an amount oflight reflected from the eye region of the user.

The method may further include detecting a cornea region of an eye ofthe user from the image.

The method may further include detecting a movement of the cornea regionby using a plurality of images consecutively generated by the DVS cameramodule based on variations in an amount of light reflected from one ormore points of the eye region.

The method may further include tracking a change in a position of thedetected cornea region, based on the plurality of images consecutivelygenerated by the DVS camera module.

The method may further include detecting a pupil region of the user fromthe image, based on the detected cornea region.

The method may further include storing a position of the detected pupilregion in a memory, and based on a change in the position of thedetected pupil region, updating the position of the detected pupilregion and storing the updated position in the memory.

The method may further include, based on the pupil region not beingdetected from the image, estimating a current position of the pupilregion, based on the position of the pupil region stored in the memory.

The method may further include determining a direction of a view of theeye, based on a position of the cornea region and the position of thepupil region, and storing the positions of the cornea region and thepupil region and the determined direction of the view in the memory.

The detecting the pupil region may include filtering a peripheral regionoutside the cornea region from the image based on the detected cornearegion.

In accordance with an aspect of the disclosure, there is provided anon-transitory computer-readable recording medium having recordedthereon a computer program for executing the above-described operationmethod(s).

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features, and advantages of certainembodiments of the disclosure will be more apparent from the followingdescription taken in conjunction with the accompanying drawings, inwhich:

FIG. 1 is a view for schematically describing an eye tracking apparatusmounted on a wearable device, according to an embodiment;

FIG. 2 is a flowchart of a method, performed by an eye trackingapparatus, of generating a dynamic vision sensor (DVS) image, accordingto an embodiment;

FIG. 3A is a view for explaining an example of changing a direction oflight under control of a scanning mirror, according to an embodiment;

FIG. 3B is a view for explaining an example of changing a direction oflight under control of a scanning mirror, according to an embodiment;

FIG. 4 is a view for explaining a difference in an amount of lightreflected at a boundary point of an cornea of a user's eye, according toan embodiment;

FIG. 5 is a flowchart of a method of detecting a boundary point of ancornea of a user's eye and an cornea region of the user's eye, accordingto an embodiment;

FIG. 6 is a view for explaining a principle in which a boundary point ofan cornea and an cornea region are detected from a DVS image, accordingto an embodiment;

FIG. 7 is a flowchart of a method of detecting a pupil region of auser's eye, according to an embodiment;

FIG. 8 is a view for explaining an example of detecting a pupil region,according to an embodiment;

FIG. 9 is a view for explaining an example in which a pupil region isdetected via natural light, according to an embodiment;

FIG. 10 is a flowchart of a method of estimating a pupil region,according to an embodiment;

FIG. 11 is a view for explaining an example in which no pupil region isdetected from a DVS image, according to an embodiment;

FIG. 12 is a view for explaining a principle in which a pupil region isestimated from a stored DVS image, according to an embodiment;

FIG. 13 is a flowchart of a method of tracking a direction of view of aneye, according to an embodiment;

FIG. 14 is a view for explaining a direction of view of a user's eye,according to an embodiment;

FIG. 15A is a view for explaining an example of a direction of view of auser's eye, according to an embodiment;

FIG. 15B is a view for explaining an example of a direction of view of auser's eye, according to an embodiment;

FIG. 16 is a flowchart of an operation method of an eye trackingapparatus, according to an embodiment.

FIG. 17 is a flowchart of an operation method, of a wearable device, ofperforming eye tracking, according to an embodiment.

FIG. 18A is a view for explaining an arrangement example of a lightsource module and a DVS camera module, according to an embodiment;

FIG. 18B is a view for explaining an arrangement example of a lightsource module and a DVS camera module, according to an embodiment;

FIG. 18C is a view for explaining an arrangement example of a lightsource module and a DVS camera module, according to an embodiment;

FIG. 18D is a view for explaining an arrangement example of a lightsource module and a DVS camera module, according to an embodiment;

FIG. 19 is a block diagram of an eye tracking apparatus according to anembodiment;

FIG. 20 is a block diagram of a wearable device including an eyetracking apparatus, according to an embodiment; and

FIG. 21 is a block diagram of a wearable device and an external device,according to an embodiment.

DETAILED DESCRIPTION

Embodiments of the disclosure are described in detail herein withreference to the accompanying drawings so that this disclosure may beeasily performed by one of ordinary skill in the art to which thedisclosure pertains. The disclosure may, however, be embodied in manydifferent forms and should not be construed as being limited to theexamples set forth herein. In the drawings, parts irrelevant to thedescription are omitted for simplicity of explanation, and like numbersrefer to like elements throughout.

Although general terms widely used at present were selected fordescribing the disclosure in consideration of the functions thereof,these general terms may vary according to intentions of one of ordinaryskill in the art, case precedents, the advent of new technologies, andthe like. Hence, the terms used herein are defined based on theirmeanings in relation to the contents of the entire specification, not bytheir simple meanings.

Throughout the disclosure, the expression “at least one of a, b or c”indicates only a, only b, only c, both a and b, both a and c, both b andc, all of a, b, and c, or variations thereof.

While such terms as “first,” “second,” etc., may be used to describevarious components, such components must not be limited to the aboveterms. The above terms are used only to distinguish one component fromanother.

The terms used in the present specification are merely used to describeparticular embodiments of the disclosure, and are not intended to limitthe scope of the disclosure. An expression used in the singularencompasses the expression of the plural, unless it has a clearlydifferent meaning in the context. Throughout the specification, when anelement is referred to as being “connected” or “coupled” to anotherelement, it may be directly connected or coupled to the other element,or may be electrically connected or coupled to the other element withintervening elements interposed therebetween. In addition, the terms“comprise” and/or “comprising” or “includes” and/or “including” whenused in this disclosure, specify the presence of stated elements, but donot preclude the presence or addition of one or more other elements.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosure (especially in the context of thefollowing claims) are to be construed to cover both the singular and theplural. Also, operations or steps of any method described herein may beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. Embodiments of the disclosureare not limited to the described order of the operations.

Thus, the expression “according to an embodiment of the disclosure” usedin the entire disclosure does not necessarily indicate the sameembodiment of the disclosure.

Embodiments of the disclosure may be described in terms of functionalblock components and various processing steps. Some or all of suchfunctional blocks may be realized by any number of hardware and/orsoftware components configured to perform the specified functions. Forexample, functional blocks according to the disclosure may be realizedby one or more microprocessors or by circuit components for a certainfunction. In addition, for example, functional blocks according to thedisclosure may be implemented with any programming or scriptinglanguage. The functional blocks may be implemented in algorithms thatare executed on one or more processors. Furthermore, the disclosuredescribed herein could employ any number of conventional techniques forelectronics configuration, signal processing and/or control, dataprocessing and the like. The words “mechanism,” “element,” “means,” and“configuration” are used broadly and are not limited to mechanical orphysical embodiments of the disclosure,

Furthermore, the connecting lines or connectors between components shownin the various figures presented are intended to represent exemplaryfunctional relationships and/or physical or logical couplings betweenthe components. Connections between components may be represented bymany alternative or additional functional relationships, physicalconnections or logical connections in a practical device.

The use of any and all examples, or exemplary language (e.g., “forexample” and “such as”) provided herein, is intended merely to betterilluminate the disclosure and does not pose a limitation on the scope ofthe disclosure unless otherwise claimed.

Moreover, no item or component is essential to the practice of thedisclosure unless the element is specifically described as “essential”or “critical”.

It will be understood by those of ordinary skill in the art that variouschanges in form and details may be made to the embodiment of thedisclosure without departing from the intrinsic characteristics of theabove descriptions.

As the disclosure allows for various changes and numerous embodiments ofthe disclosure, particular embodiments of the disclosure will beillustrated in the drawings and described in detail in the writtendescription. However, this is not intended to limit the disclosure toparticular modes of practice, and it is to be appreciated that allchanges, equivalents, and substitutes that do not depart from the spiritand technical scope are encompassed in the disclosure. Thus, it shouldbe understood that the disclosed embodiments of the disclosure should beconsidered in a descriptive sense only and not for purposes oflimitation.

While one or more example embodiments of the disclosure have beendescribed with reference to the figures, it will be understood by one ofordinary skill in the art that various changes in form and details maybe made therein without departing from the spirit and scope as definedby the following claims.

The terms “unit”, “-er (-or)”, and “module” when used in thisspecification refers to a unit in which at least one function oroperation is performed, and may be implemented as hardware, software, ora combination of hardware and software.

The terms “ . . . or(er)”, “ . . . interface”, and “ . . . module” maybe stored in an addressable storage medium and may be implemented by aprogram that may be executed by a processor.

For example, the “ . . . or(er)”, “ . . . interface”, and “ . . .module” may be implemented by object-oriented software components, classcomponents, and task components, and processes, functions, attributes,procedures, subroutines, segments of a program code, drivers, firmware,a micro code, a circuit, data, a database, data structures, tables,arrays, and variables.

In this specification, a recitation “A may include one of a1, a2, anda3” broadly means that an exemplary element that may be included in theelement A is a1, a2, or a3.

The above recitation does not necessarily imply that an element capableof constituting the element A is limited to a1, a2 or a3. Thus, itshould be noted that elements that may constitute A are not exclusivelyinterpreted in the sense that they exclude other elements notillustrated except a1, a2 and a3.

In addition, the above recitation means that A may include a1, includea2, or include a3. The above recitation does not imply that elementsconstituting A are necessarily determined within a given set. Forexample, it should be noted that the above recitation does notnecessarily construe that a1, a2, or a3 selected from a set includinga1, a2, and a3 constitutes component A.

The disclosure will now be described more fully with reference to theaccompanying drawings, in which exemplary embodiments of the disclosureare shown.

FIG. 1 is a view for schematically describing an eye tracking apparatus100 mounted on a wearable device 1 according to an embodiment.

The eye tracking apparatus 100 according to an embodiment may include afirst eye tracker 101 for tracking the view of a user's left eye and asecond eye tracker 102 for tracking the view of a user's right eye.

FIG. 1 illustrates an example where the eye tracking apparatus 100 ofFIGS. 1 and 19 (hereinafter, the eye tracking apparatus 100) is mountedon the wearable device 1 of FIGS. 1 and 20 (hereinafter, the wearabledevice 1).

According to an embodiment, the wearable device 1 may be a glasses-typewearable device, for example, an augmented reality (AR) apparatus forproviding an AR image or a virtual reality (VR) apparatus for providinga VR image. However, embodiments are not limited thereto.

Because the first eye tracker 101 and the second eye tracker 102 havethe same structure and operate in the same manner, FIG. 1 will now bedescribed with respect to the first eye tracker 101. That is, thefollowing description provided with respect to the first eye tracker 101applies to the second eye tracker 102.

The first eye tracker 101 according to an embodiment may include a lightsource module 301 of FIGS. 1 and 19 (hereinafter, the light sourcemodule 301) and a dynamic vision sensor (DVS) camera module 304 of FIGS.1 and 19 (hereinafter, the DVS camera module 304). The second eyetracker 102 according to an embodiment may include the light sourcemodule 301 and the DVS camera module 304.

The light source module 301 according to an embodiment may provide lightto a user's eye.

The light source module 301 may include a light source 302 of FIG. 19(hereinafter, the light source 302) configured to provide light, and ascanning mirror 303 of FIG. 19 (hereinafter, the scanning mirror 303)configured to control the direction of the light provided by the lightsource 302.

The light source 302 according to an embodiment may include, forexample, an infrared light-emitting diode (IR LED).

The scanning mirror 303 according to an embodiment may control thedirection of the light (for example, IR light) provided by the lightsource 302 such that the light travels toward an eyeball region 330including a cornea 310 of the user's eye.

The scanning mirror 303 may include a structure capable of mechanicallychanging a reflection angle such that the light provided by the lightsource 302 is reflected toward the user's eye, and may scan the eyeballregion 330 including the cornea 310 by using the light provided by thelight source 302.

According to an embodiment, as the light is provided to the eyeballregion 330 including the cornea 310 of the user's eye under the controlof the scanning mirror 303, light may be reflected from by the user'seye.

The light source module 301 and the DVS camera module 304 according toan embodiment may be arranged such that the light provided by the lightsource module 301 is projected toward the user's eye under the controlof the scanning mirror 303 and the reflected light reflected by theuser's eye travels toward the DVS camera module 304. According to anembodiment, various arrangement examples of the light source module 301and the DVS camera module 304 mounted on the wearable device 1 will bedescribed later with reference to FIGS. 18A through 18D.

The DVS camera module 304 according to an embodiment may include acamera module implemented as a DVS.

The DVS camera module 304 may generate image data when there is a changein the amount of light sensed by the DVS camera module 304 in units ofpixels. The DVS camera module 304 generates image data according to avariation in light sensed and does not perform data processing whenthere is no variation in the sensed light.

A DVS image obtained by the DVS camera module 304 may include pixel datarepresenting a change in light sensed according to a movement of auser's eye (e.g., movement of the user's pupil).

The DVS camera module 304 according to an embodiment may sense lightreflected by the eyeball region 330 including the cornea 310 of theuser's eye, and may generate the DVS image, based on the change in theamount of the light sensed by the DVS camera module 304.

According to an embodiment, the eye tracking apparatus 100 may track theview of the user's eye by detecting a cornea region and a pupil regionfrom the generated DVS image.

The DVS camera module 304 may react to even a small amount of light, maybe resilient or sensitive to an illumination change, and may processdata with low power and at high speed. Accordingly, the eye trackingapparatus 100 according to an embodiment may perform real-time eyetracking by using the light source module 301 and the DVS camera module304.

FIG. 2 is a flowchart of a method, performed by the eye trackingapparatus 100, of generating the DVS image, according to an embodiment.FIG. 3A is a view for explaining an example of changing the direction oflight under the control of a scanning mirror according to an embodiment.FIG. 3B is a view for explaining an example of changing the direction oflight under the control of a scanning mirror according to an embodiment.FIG. 4 is a view for explaining a difference of the amount of lightreflected at a boundary point of a cornea, according to an embodiment.

FIGS. 3A through 4 are views used to explain the method of FIG. 2.

In operation S201 of FIG. 2, the light source module 301 of FIG. 19included in the eye tracking apparatus 100 may provide light from thelight source 302 of FIG. 19 toward an eye region of a user whilechanging the direction of the light at intervals of a predetermined timeperiod under the control of the scanning mirror 303 of FIG. 19.

According to an embodiment, the light source module 301 may include thelight source 302 of FIG. 19 configured to provide light, and thescanning mirror 303 of FIG. 19 configured to control the direction ofthe light provided by the light source 302.

According to an embodiment, the light (for example, IR light) providedby the light source 302 of the light source module 301 may be projectedtoward the eyeball region 330 including the cornea region 310, whilemoving in a predetermined pattern (for example, moving along alongitudinal straight line or along a transverse straight line) at apredetermined speed (for example, 180°/ms) under the control of thescanning mirror 303.

According to an embodiment, the scanning mirror 303 may control tochange a reflection angle of light such that the light provided by thelight source 302 is reflected toward one point of the user's eye. Basedon a predetermined light changing pattern, the scanning mirror 303 maycontrol the light provided by the light source 302 to travel towardanother point of the user's eye by changing the reflection angle of thelight, so that the entire region of the user's eye may be scanned.According to an embodiment, the light source module 301 may closely scanthe entire eyeball region 330 at very close intervals, under the controlof the scanning mirror 303. The light source module 301 may projectlight toward the eyeball region 330 while changing the direction of thelight in a predetermined light changing pattern for scanning the entireeyeball region 330. Here, the eyeball region 330 refers to a region ofan eyeball that is exposed.

According to an embodiment, the pattern in which the direction of thelight provided by the light source module 301 is changed may include atleast one of a pattern in which the light moves along a longitudinalstraight line or a pattern in which the light moves along a transversestraight line.

Referring to FIG. 3A, for example, the direction of the light providedby the light source module 301 may have a pattern in which, when thelight is started to be projected from above a left upper side of theeyeball region 330, the direction in which light is projected changesrightwards along a transverse straight line 305, and then reaches aright upper side of the eyeball region 330, the projection of the lightmoves downwards along a column, as indicated by an arrow 306, and thenmoves leftwards along a transverse straight line 307. However, this ismerely an example and the disclosure is not limited thereto.

For example, the pattern in which the direction of the light provided bythe light source module 301 is changed may be a pattern in which lightstarts to be projected from above the left upper side of an eyeballregion, the direction of projection of the light moves downwards in alongitudinal straight line, then moves rightwards along a row, andupwards in a longitudinal straight line.

Referring to FIG. 3B, the pattern in which the projection direction ofthe light provided by the light source module 301 is changed may includea Lissajous curve pattern.

While the projection direction of the light provided by the light sourcemodule 301 is being changed in the Lissajous curve pattern, the lightmay be projected toward the eye region of the user. Here, the term “eyeregion” may be interchangeably used with the term “eyeball region”,which refers to a region of an eyeball that is exposed.

The pattern in which the projection direction of light is changed is notlimited to the examples of FIGS. 3A and 3B, and may include any patternthat may be used to scan the entire eyeball region 330.

In operation S202 of FIG. 2, the first eye tracker 101 may generate theDVS image, based on a variation in the amount of light reflected by theeye region of the user, by using the DVS camera module 304.

According to an embodiment, the DVS image may be generated to determinewhether there is a movement of the eye region of the user, based onwhether there is a variation in the amount of light measured at one ormore points of the eye region of the user for each of consecutiveframes. The DVS image may include pixel data representing a change inthe amount of light measured for each of the consecutive frames whenthere is the change.

Referring to FIG. 4, because the cornea 310 of the user protrudes fromthe eyeball region 330 according to the eye structure, when the amountof light reflected by a boundary point 403 of the protruding cornea 310is compared with the amounts of light beams reflected by points 401 and402 other than the cornea 310, there is a sharp difference therebetweendue to a difference between the refractive indexes of incident lightbeams.

For example, the amount of light reflected by the boundary point 403 ofthe cornea 310 may be sharply reduced compared with the amounts of lightbeams reflected by the points 401 and 402 other than the cornea 310.

According to an embodiment, the DVS camera module 304 may sense a sharpdifference between the amount of light reflected by the point 402 of theeyeball region 330 and the amount of light reflected by the boundarypoint 403 of the cornea 310, which is adjacent to the point 402, as aprojection direction of the light provided by the light source module301 changes over time, and may generate a DVS image representing achange in the light amount.

According to an embodiment, even when the user's eye actually does notmove, the DVS camera module 304 scans the user's eye with the lightprovided by the light source module 301 while continuously changing thedirection of the light over time, and thus may sense a differencebetween the amount of the reflected light at the boundary point of thecornea and the amount of the reflected light at a region outside thecornea. Accordingly, the DVS camera module 304 may generate a DVS imagerepresenting a change in the amount of light measured for each ofconsecutive frames, in units of pixels.

According to an embodiment, the wearable device 1 including the eyetracking apparatus 100 may track the view of the user's eye by detectinga cornea region from the DVS image.

A method of detecting a cornea region and a pupil region from a DVSimage will now be described with reference to FIGS. 5 through 12.

FIG. 5 is a flowchart of a method of detecting a boundary point of acornea of a user's eye and a cornea region of the user's eye, accordingto an embodiment. FIG. 6 is a view for explaining a principle in whichthe boundary point of the cornea and the cornea region are detected froma DVS image, according to an embodiment. FIG. 6 is a view used toexplain the method of FIG. 5.

According to an embodiment, the eye tracking apparatus 100 may provide agenerated DVS image to a processor 120 of FIG. 20 (hereinafter, aprocessor 120) included in the wearable device 1 of FIG. 20.

In operation S501 of FIG. 5, the processor 120 may detect the boundarypoint of the cornea of the user's eye from the DVS image.

According to an embodiment, an operation of scanning the entire eyeballregion 330 while changing the direction of the light provided by thelight source module 301 in, for example, the pattern of FIG. 3A or 3B,may be performed. In this scanning operation, a substantial differencebetween the amount of reflected light when light is projected to onepoint outside the cornea 310 adjacent to the boundary point of thecornea and the amount of reflected light when light is projected to theboundary point of the cornea 310 may be detected. Accordingly, a DVSimage 601 may be generated such that a boundary point 602 of the cornea,from which a substantial variation in the light amount is sensed, isdistinguishable from a peripheral region.

According to an embodiment, even when there is no movement of the user'seye, the boundary point 602 of the cornea may be detected from the DVSimage 601 due to a change in the amount of reflected light at theboundary point of the cornea 310.

In operation S502 of FIG. 5, the processor 120 may detect the cornearegion of the user's eye, based on the detected boundary point of thecornea.

For example, referring to FIG. 6, the processor 120 may determine aninternal region of the detected boundary point 602 of the cornea to bethe cornea region.

According to an embodiment, as an operation of projecting light to theeye region while changing the direction of light over time under thecontrol of the scanning mirror 303 is repeated, the processor 120 maytrack a change in the position of the detected cornea region.

Accordingly, the processor 120 may track a change in the view of theuser's eye, based on the change in the position of the cornea.

FIG. 7 is a flowchart of a method of detecting a pupil region of auser's eye, according to an embodiment. FIG. 8 is a view for explainingan example of detecting the pupil region, according to an embodiment.FIG. 9 is a view for explaining an example in which the pupil region isdetected by natural light, according to an embodiment.

FIGS. 8 and 9 are schematic views used to explain the method of FIG. 7.

According to an embodiment, the eye tracking apparatus 100 may provide agenerated DVS image to the processor 120 of the wearable device 1 ofFIG. 20.

In operation S701 of FIG. 7, the processor 120 may detect the pupilregion of the user's eye from the DVS image, based on the cornea regiondetected in operations S501 and S502 of FIG. 5.

Referring to FIG. 8, according to an embodiment, the processor 120 maydetect, as pupil regions 803 and 806, regions distinguishable fromperipheral regions from among the internal regions of cornea regions 802and 805 detected from DVS images 801 and 804.

According to an embodiment, because a pupil region 820 within an eyeballregion has a dark black color, when light is projected onto the pupilregion 820, the pupil region 820 may absorb more light, compared with aperipheral region outside the pupil region 820. Accordingly, the amountof reflected light at the pupil region 820 is greatly different from theamount of reflected light when light is projected onto the peripheralregion outside the pupil region 820. Accordingly, as a scan operation ofthe light source module 301 is repeated, the pupil regions 803 and 806may be distinguishable from the peripheral regions, on the DVS images801 and 804 generated by the DVS camera module 304.

Referring to FIG. 8, when a cornea 810 and the pupil region 820 of theuser's eye move left and right, the DVS camera module 304 mayconsecutively generate a plurality of DVS images, as an operation ofscanning the entire region of the user's eye is repeated. Theconsecutively-generated plurality of DVS images may represent movementof the user's eye according to changes in the positions of the cornea810 and the pupil region 820 of the user's eye.

The DVS camera module 304 may measure a difference of the light amountfor each pixel when a current frame is compared with a previous framefrom among consecutively captured frames. The DVS camera module 304 maygenerate a (+) signal in a pixel in which a measured light amount issensed to increase, e.g., from dark to bright, and thus may display apoint having an increased light amount in, for example, a white color.The DVS camera module 304 may generate a (−) signal in a pixel in whicha measured light amount is sensed to decrease, e.g., from bright todark, and thus may display a point having a decreased light amount in,for example, a black color. Accordingly, a region of the DVS image wherean object (e.g., a pupil or a cornea) newly appears according to amovement of the object may be displayed in white, and a region of theDVS image where the object (e.g., a pupil or a cornea) disappearsaccording to the movement of the objection may be displayed in black.

According to an embodiment, a pixel region of the DVS image where the(+) signal or the (−) signal is not generated may be displayed in, forexample, a gray color to be easily distinguishable from a black color ora white color. However, the colors in which pixel regions of the DVSimage are displayed according to the (+) signal or the (−) signal arenot limited to these examples.

For example, the pixel where the (+) signal is generated being displayedin white and the pixel where the (−) signal is generated being displayedin black is merely an example, and thus embodiments are not limitedthereto. According to settings of the eye tracking apparatus 100, pixelsof the DVS image may be displayed in different colors such that thepixel where the (+) signal is generated, the pixel where the (−) signalis generated, and the pixel where no (+) or (−) signal is generated maybe displayed in various colors that are easily distinguishable from oneanother.

Referring to FIG. 8, for example, the DVS image 801, which is generatedwhen the user's eye moves to left, may indicate that the cornea region810 and the pupil region 820 of the user's eye move leftwards, bydisplaying, in white, pixels corresponding to respective left portionsof the cornea region 802 and the pupil region 803 (that is, where thecornea region 802 and the pupil region 803 newly appear due to themovement of the eye and thus the (−) signal is generated) anddisplaying, in black, pixels of the DVS image 801 corresponding torespective right portions of the cornea region 802 and the pupil region803 (that is, where the cornea region 802 and the pupil region 803disappear due to the movement of the eye and thus the (+) signal isgenerated). In addition, the DVS image 804, which is generated when theuser's eye moves to right, may indicate that the cornea region 810 andthe pupil region 820 of the user's eye move rightwards, by displaying,in black, pixels corresponding to respective left portions of the cornearegion 805 and the pupil region 806 (that is, where the cornea region805 and the pupil region 806 disappear and thus the (+) signal isgenerated) and displaying, in white, pixels corresponding to respectiveright portions of the cornea region 805 and the pupil region 806 (thatis, where the cornea region 805 and the pupil region 806 newly appearand thus the (−) signal is generated).

Referring to FIG. 9, because natural light (for example, sunlight 901 ordisplay light 902) may be incident upon the front of the user's eye in anormal environment of the user, a difference in the light amount ofreflected light due to natural light may be generated between a pupilregion 920 having a dark black color within the eyeball region 330 and aperipheral region other than the pupil region 920 within the eyeballregion 330.

Accordingly, as shown in FIG. 8, when the user's eye moves, namely, whenthe pupil region 820 moves, the DVS camera module 304 may sense amovement of the pupil region 820 having a dark black color, and the DVSimages 801 and 804 may be generated such that the pupil regions 803 and806 are distinguishable from other regions.

In operation S702 of FIG. 7, the processor 120 may store a position ofthe detected pupil region in a memory 130 of FIG. 20.

According to an embodiment, as the position of the detected pupil regionchanges, the processor 120 may update a changed position and store thechanged position in the memory 130 of FIG. 20. Accordingly, theprocessor 120 may determine a position of the pupil of the user that isa position at the most recent time point from a current time point.

According to an embodiment, while an operation of providing light to theeye region while changing the direction of the light under the controlof the scanning mirror 303 is performed, the processor 120 may track achange in the position of the pupil by updating a changed position ofthe detected pupil region and storing the changed position in the memory130 of FIG. 20.

Accordingly, the processor 120 of the wearable device 1 may track achange in the view of the user's eye, based on the change in theposition of the pupil.

FIG. 10 is a flowchart of a method of estimating a pupil region,according to an embodiment. FIG. 11 is a view for explaining an examplein which no pupil regions are detected from a DVS image, according to anembodiment. FIG. 12 is a view for explaining a principle in which apupil region is estimated from a stored DVS image, according to anembodiment.

FIGS. 11 and 12 are schematic views used to explain the method of FIG.10.

According to an embodiment, the eye tracking apparatus 100 may provide agenerated DVS image to the processor 120.

In operation S1001 of FIG. 10, as the position of the detected pupilregion changes, the processor 120 may update the changed position of thepupil region and store the changed position in the memory 130 of FIG.20.

According to an embodiment, as the position of the detected pupil regionis changed during a repetitive scan operation of the light source module301 on the eyeball region, the processor 120 may update a changedposition of the pupil region and store the changed position in thememory 130 of FIG. 20.

Accordingly, the processor 120 may track a change in the position of thepupil.

In operation S1002 of FIG. 10, as no pupil regions are detected from theDVS image, the processor 120 may estimate a current position of thepupil region, based on the position of the pupil region stored in thememory 130 of FIG. 20.

According to an embodiment, when no pupil regions are detected from theDVS image, the processor 120 may determine the current position of thepupil region, based on a position of the pupil region that has beenupdated and stored most recently from a current time point.

Referring to FIG. 11, for example, when the user's eye moves leftwards,a first DVS image 1101 and a second DVS image 1104 may be displayed suchthat a first cornea region 1102 is distinguishable from a first pupilregion 1103 and a second cornea region 1105 is distinguishable from asecond pupil region 1106. However, when the movement of the user's eyestops, a third cornea region 1108 may be displayed on a third DVS image1107, but no third pupil region may be displayed, which will bedescribed in detail below.

The DVS camera module 304 may generate image data, based on a change insensed light. When the user's eye moves, the eye tracking apparatus 100may generate a DVS image due to a change in the amount of light measuredat each point on the eye region.

When the user's eye does not move, a difference in reflected light maybe generated at the boundary point of the cornea due to the lightprovided by the light source module 301 while the direction of the lightis being changed, and thus the boundary point of the cornea may bedistinguishable on the DVS image. Because a pupil region having a darkblack color absorbs a large amount of light provided by the light sourcemodule 301, compared with the peripheral region outside the pupilregion, a difference in reflected light is generated, and thus the pupilregion may be displayed distinguishably on the DVS image.

However, when the user wears the wearable device 1 of FIG. 1 includingthe eye tracking apparatus 100, light from the light source module 301may be incident upon the user's eye from a direction that is lateral tothe user's eye. Thus, in this usage environment, a change in the amountof reflected light may be generated at the boundary point of theprotruding cornea, but a change in the amount of reflected light may notbe generated or only a little change may be generated on a pupil regionthat does not protrude as the cornea region. According to an embodiment,the DVS camera module 304 may not generate a (+) or (−) signal accordingto a change in the light amount at a point where a measured change inthe light amount is less than a predetermined threshold. Accordingly,the DVS image may not include any indication at the point where themeasured change in the light amount is less than the predeterminedthreshold.

For example, sensitivity of a light amount change for generating the (+)or (−) signal in the DVS camera module 304 is set to be high, namely, athreshold of a light amount change for generating the (+) or (−) signalis set to be low, the pupil region may be distinguishably displayed inthe DVS image. However, when the sensitivity of the light amount changefor generating the (+) or (−) signal in the DVS camera module 304 is setto be low, namely, the threshold of a light amount change for generatingthe (+) or (−) signal is set to be high, the pupil region may not bedisplayed in the DVS image.

For example, because the DVS camera module 304 generates the (+) or (−)signal according to a change in the light amount, the pupil region maynot be sufficiently distinguishable on the DVS image or not shown on theDVS image.

According to an embodiment, when the pupil region is not shown on theDVS image or is shown vaguely with a definition less than apredetermined standard, the processor 120 may determine that no pupilregion is detected.

According to an embodiment, when the processor 120 determines that nopupil region is detected or the pupil region is not clearlydistinguished from the DVS image, the processor 120 may estimate thecurrent position of the pupil region, based on a position of the pupilregion that is stored at a time point closest to a current time.

Referring to FIGS. 11 and 12, when the positions of the first and secondpupil regions 1103 and 1106 on first and second DVS images consecutivelyproduced while the user's eye moved leftwards are stored in a memory,the position of a third pupil region 1201 on the third DVS image 1107may be estimated based on the positions of the first and second pupilregions 1103 and 1106 and the positions of the first, second, and thirdcornea regions 1102, 1105, and 1108.

According to an embodiment, even when only the cornea region is shown onthe DVS image and the position of the pupil region is not shown or isshown unclearly on the DVS image, the processor 120 may estimate thecurrent position of the pupil region by using the position of the pupilregion stored in the memory 130 of FIG. 20.

Accordingly, the processor 120 may track a change in the view of theuser's eye by tracking the positions of the cornea and the pupil of theuser's eye.

FIG. 13 is a flowchart of a method of tracking a direction of a view ofan eye, according to an embodiment. FIG. 14 is a view for explaining adirection of a view of a user's eye, according to an embodiment. FIGS.15A and 15B are views for explaining an example of the direction of aview of the user's eye, according to an embodiment.

FIGS. 14 through 15B are views used to explain the method of FIG. 13.

According to an embodiment, the eye tracking apparatus 100 may provide agenerated DVS image to the processor 120.

According to an embodiment, the processor 120 may determine a directionof a view of an eye, based on the positions of a cornea region and apupil region of the eye. As a scan operation of the DVS camera module304 is repeated, the processor 120 may track a change in the directionof view of the eye.

In operation S1301 of FIG. 13, the processor 120 may determine thedirection of the view of the eye, based on the positions of a cornearegion and a pupil region of the eye.

According to an embodiment, the direction of the view may refer to adirection that is viewed by the user's eye. According to an embodiment,eye tracking may be performed by tracking the change in the direction ofthe view according to a movement of the user's eye.

FIG. 14 is a view for explaining a direction of a view of a user's eye,according to an embodiment. FIG. 14 illustrates a three-dimensional (3D)eyeball model of a user.

The processor 120 may determine the direction of a view of a user's lefteye by using the first eye tracker 101 of FIG. 1, and may determine thedirection of the view of the user's right eye by using the second eyetracker 102 of FIG. 1.

For example, the processor 120 may determine the direction of the view,based on an average eyeball model of a person. The eyeball model may bemodeled by assuming that an eyeball 3100 of a human has a sphericalshape and the eyeball 3100 ideally rotates according to the direction ofthe view.

In FIG. 14, d denotes a distance between a center 3150 of a user's eyeand a virtual screen 3200, and a case where the user's eye gazes at apoint O of the virtual screen 3200 is illustrated. In FIG. 14, when theuser's eye that gazes at the point O changes the view to gaze at a viewpoint (x, y) on the virtual screen 3200, a denotes an angle of rotationof the user's eye in the x-axis direction in accordance with the changeof the view, and β denotes an angle of rotation of the user's eye in they-axis direction in accordance with the change of the view. In FIG. 14,r denotes the radius of a sphere assuming that the user's eye has ashape of the sphere.

According to an embodiment, the processor 120 may calculate thedirection of the view, based on the positions of the cornea 310 and thepupil 320 detected from the DVS image obtained from the eye trackingapparatus 100.

According to an embodiment, the direction of the view may refer to adirection that is viewed by the user's eye. The direction of the viewmay be determined based on the center 3150 of the user's eye and thepositions of the cornea 310 and the pupil 320. The processor 120according to an embodiment may calculate a changed direction of theview, based on positions of the cornea and the pupil changed from aplurality of DVS images obtained consecutively.

The processor 120 may calculate degrees of rotation (e.g., α and β) ofthe user's eye (e.g., the left eye) from the consecutively-obtainedplurality of DVS images, and the processor 120 may use the degrees ofrotation α and β of the user's eye to calculate a two-dimensional (2D)coordinate value (x, y) to which the direction of the view of the user'seye on the virtual screen 3200 has been changed.

For example, the 2D coordinate value (x, y) may be expressed as shown inEquation 1 below.x=d·tan α,y=d·sec α·tan β,  [Equation 1]

Referring to FIG. 15A, for example, a gaze point G1, 1501 wheredirections of views of the user's both eyes are met may exist on avirtual screen 3201 on an actual space viewed by a user of the wearabledevice 1.

Referring to FIG. 15B, for example, a gaze point G2, 1502 where thedirections of views of the user's both eyes are met may exist on ascreen 3202 on a display viewed by the user of the wearable device 1.

According to an embodiment, the processor 120 may estimate a coordinatevalue of a gaze point by using binocular disparity of both eyes, thedirection of the view of the left eye, and the direction of the view ofthe right eye.

In operation S1302 of FIG. 13, the processor 120 may store the positionsof the cornea region and the pupil region and the determined directionof the view in the memory 130 of FIG. 20.

According to an embodiment, the processor 120 may store the positions ofthe cornea region and the pupil region detected using the method ofFIGS. 1 through 12 in the memory 130 of FIG. 20. The processor 120 maystore the direction of the view determined based on the positions of thecornea region and the pupil region, in the memory 130 of FIG. 20.

In operation S1303 of FIG. 13, the processor 120 may track a change inthe direction of the view, as a scan operation of the DVS camera module304 is repeated.

According to an embodiment, the eye tracking apparatus 100 may repeat ascan operation of the DVS camera module 304. According to an embodiment,the scan operation may refer to an operation of providing light from thelight source 302 toward an eye region while changing the direction ofthe light under the control of the scanning mirror 303.

The eye tracking apparatus 100 may provide, to the processor 120, aplurality of DVS images consecutively produced by the DVS camera module304 as a scan operation is repeated. The processor 120 may determine thepositions of the cornea region and the pupil region from each of theplurality of DVS images, and may determine the direction of the eye'sview.

The processor 120 may store the cornea region and the pupil region andthe determined direction of the eye's view, in the memory 130 of FIG.20. The processor 120 may track a change in the direction of the view,based on changes in the positions of the cornea region and the pupilregion that are sequentially stored.

FIG. 16 is a flowchart of an operation method of the eye trackingapparatus 100, according to an embodiment.

In operation S1601 of FIG. 16, the eye tracking apparatus 100 may drivethe DVS camera module 304.

The DVS camera module 304 according to an embodiment may sense lightreflected by an eye region of a user, and may generate a DVS image,based on a change in the amount of sensed light.

In operation S1602 of FIG. 16, the eye tracking apparatus 100 mayprovide light from the light source 302 toward the eye region of theuser.

The scanning mirror 303 included in the light source module 301 of FIG.1 according to an embodiment may control the direction of the light (forexample, IR light) provided by the light source 302 such that the lighttravels toward the eye region. As the light is provided to the eyeregion under the control of the scanning mirror 303, light reflectedfrom the user's eye is detected.

According to an embodiment, the light source module 301 of FIG. 1 mayprovide light toward one point on the eye region, without changing thedirection of the light over time, under the control of the scanningmirror 303.

In operation S1603 of FIG. 16, the eye tracking apparatus 100 maydetermine whether a DVS image has been generated.

According to an embodiment, as the user's eye moves, the DVS cameramodule 304 may generate the DVS image, based on a difference of theamounts of light beams sensed on the boundary point of the cornea of theuser's eye and a difference of the amounts of light beams sensed on thepupil region.

When it is determined that a DVS image has been generated, the eyetracking apparatus 100 may provide light toward the eye region, withoutchanging the direction of the light over time (operation S1602). Basedon the DVS image, tracking of an eye may be performed.

According to an embodiment, when there is a movement of the user's eye,the DVS camera module 304 generates the DVS image without an operationin which the scanning mirror 303 controls the direction of light to bechanged at intervals of a predetermined time period, and thusdirection-change control of the scanning mirror 303 may not be driven.

Accordingly, the eye tracking apparatus 100 may be driven with lowpower, and power consumption thereof may be minimized.

According to an embodiment, when there is no movement of the user's eye,the DVS camera module 304 is unable to sense a change in the lightamount due to a movement, and thus may generate no DVS images.

In operation S1604 of FIG. 16, based on a determination that no DVSimages are generated, the eye tracking apparatus 100 may provide thelight provided by the light source 302 toward the eye region of the userwhile changing the direction of the light at intervals of apredetermined time period under the control of the scanning mirror 303.

According to an embodiment, when it is determined that no DVS images aregenerated, the eye tracking apparatus 100 may drive the scanning mirror303 so that the direction of the light is changed at intervals of thepredetermined time period.

According to an embodiment, even when there is no movement of the user'seye, the DVS camera module 304 may sense a sharp variation in the amountof reflected light reflected by the boundary point of the cornea, as aprojection direction of the light provided by the light source module301 changes over time, and may generate a DVS image representing thechange in the light amount.

FIG. 17 is a flowchart of an operation method of the wearable device 1,according to an embodiment.

In operation S1701 of FIG. 17, the processor 120 of the wearable device1 may obtain a DVS image.

According to an embodiment, the eye tracking apparatus 100 may generatea DVS image. The generation of the DVS image by the eye trackingapparatus 100, according to an embodiment, may be performed based ondescription with reference to FIGS. 2 through 4.

The eye tracking apparatus 100 may provide the generated DVS image tothe processor 120.

In operation S1702 of FIG. 17, the processor 120 of the wearable device1 may detect a cornea region of an eye from the obtained DVS image. Thedetection of the cornea region of the eye from the DVS image by theprocessor 120 according to an embodiment may be performed based ondescription with reference to FIGS. 5 and 6.

According to an embodiment, the processor 120 may filter out a DVS imagefrom which the cornea region has not been detected, from the obtainedDVS image. For example, a DVS image obtained when user's eyes are closedmay be filtered out.

In operation S1703 of FIG. 17, the processor 120 of the wearable device1 may filter out a peripheral region outside the cornea region from theobtained DVS image.

According to an embodiment, the DVS image generated by the DVS cameramodule 304 may include not only the cornea region of the eye but also aregion other than an eyeball, such as an eyelid region or an eyelashregion. Because the processor 120 may track the view of the eye, basedon the position of the cornea region, the peripheral region outside thecornea region may be unnecessary for eye tracking. Accordingly, afterdetecting the cornea region from the DVS image generated by the DVScamera module 304, the processor 120 may filter out the peripheralregion outside the cornea region.

According to an embodiment, there may exist a case where the detectedcornea region includes only a partial region of the cornea, for example,a case where a portion of the cornea is hidden by an eyelid or aneyelash of the eye and thus the entire region of the cornea is notdetected. In this case, after filtering out the peripheral regionoutside the cornea region, such as the eyelid region or the eyelashregion, from the DVS image, the processor 120 may estimate a remainingnot-shown partial region of the cornea region, based on a shown partialregion of the cornea region.

According to an embodiment, the processor 120 may quickly and accuratelydetect a pupil region within the cornea region, based on the position ofthe cornea region, by filtering out the peripheral region outside thecornea region.

In operation S1704 of FIG. 17, the processor 120 of the wearable device1 may detect a pupil region of the eye. The detection of the pupilregion of the eye from the DVS image by the processor 120 according toan embodiment may be performed based on description with reference toFIGS. 7 through 12.

In operation S1705 of FIG. 17, the processor 120 of the wearable device1 may track the view of the eye, based on the positions of the cornearegion and the pupil region. The tracking of the view of the eye by theprocessor 120 according to an embodiment may be performed based ondescription with reference to FIGS. 13 through 15B.

FIGS. 18A through 18D are views for explaining various arrangementexamples of a light source module and a DVS camera module, according toan embodiment. According to an embodiment, the first eye tracker 101 ofFIG. 1 for tracking the view of a user's left eye and the second eyetracker 102 of FIG. 1 for tracking the view of a user's right eye may bemounted on the glasses-type wearable device 1 of FIG. 1.

Each of the first eye tracker 101 and the second eye tracker 102according to an embodiment may include the light source module 301 ofFIG. 1 and the DVS camera module 304 of FIG. 1.

Because the first eye tracker 101 and the second eye tracker 102 havethe same structure and operate in the same manner, FIGS. 18A through 18Dwill now be described based on the first eye tracker 101 as arepresentative example.

According to an embodiment, the light provided by the light sourcemodule 301 of FIG. 1 may be projected toward the user's eye, andreflected light reflected by the user's eye may be incident upon the DVScamera 304 of FIG. 1.

Accordingly, in order for the light provided by the light source module301 mounted on the glasses-type wearable device 1 of FIG. 1 to beprojected toward the user's eye and for the reflected light reflected bythe user's eye to travel toward the DVS camera module 304, the lightsource module 301 and the DVS camera module 304 may be arranged to be acertain distance spaced apart from each other with the user's eyetherebetween when the user has worn the glasses-type wearable device 1of FIG. 1.

According to an embodiment, the light source module 301 and the DVScamera module 304 may be built in a frame region of the glasses-typewearable device 1 of FIG. 1.

FIGS. 18A through 18D illustrate a frame of the glasses-type wearabledevice 1 on which the first eye tracker 101 of FIG. 1 for tracking theview of the left eye has been mounted.

As shown in FIG. 18A, a first light source module 301 a may be arrangedon a left lower portion of a frame 1 a, and a first DVS camera module304 a may be arranged on a right lower portion of the frame 1 a.

According to an embodiment, as the first light source module 301 a isarranged on a lower portion of the frame 1 a, interference by theeyelash when light is projected from the first light source module 301 atoward the eyeball region of the user may be minimized.

According to an embodiment, as the first DVS camera module 304 a isbuilt on the lower portion (for example, a nose support portion) of theframe 1 a, a space may be secured, and thus an influence of the eyetracking apparatus upon the outer appearance of the glasses-typewearable device 1 may be minimized.

As another example, as shown in FIG. 18B, a second light source module301 b may be arranged on a left upper portion of a frame 1 b, and asecond DVS camera module 304 b may be arranged on a right upper portionof the frame 1 b.

According to an embodiment, when the second DVS camera module 304 b isbuilt on an upper portion of the frame 1 b, even when the upper frame ofthe frame 1 b becomes thicker, disturbance in the view by a frame whenthe user has worn the glasses-type wearable device 1 may be minimized.

As another example, as shown in FIG. 18C, a third light source module301 c may be arranged on a left lower portion of a frame 1 c, and athird DVS camera module 304 c may be arranged on a right upper portionof the frame 1 c.

As another example, as shown in FIG. 18D, a fourth light source module301 d may be arranged on a left upper portion of a frame 1 d, and afourth DVS camera module 304 d may be arranged on a right lower portionof the frame 1 d.

According to an embodiment, when the fourth light source module 301 d isbuilt on an upper portion of the frame 1 d, even when the upper frame ofthe frame 1 d becomes thicker, disturbance in the view by a frame whenthe user has worn the glasses-type wearable device 1 may be minimized.

FIGS. 18A through 18D are views for explaining examples in which a lightsource module and a DVS camera module may be arranged, but embodimentsare not limited thereto.

FIG. 19 is a block diagram of the eye tracking apparatus 100 accordingto an embodiment.

As shown in FIG. 19, the eye tracking apparatus 100 according to anembodiment may include the light source module 301 and the DVS cameramodule 304, the light source module 301 including the light source 302and the scanning mirror 303.

More or less components than those illustrated in FIG. 19 may beincluded in the eye tracking apparatus 100.

The light source module 301 according to an embodiment may include thelight source 302 and the scanning mirror 303.

The light source 302 according to an embodiment may provide light. Forexample, the light source 302 may include an IR LED.

The scanning mirror 303 according to an embodiment may control aprojection direction of the light (for example, IR light) provided bythe light source 302.

According to an embodiment, the light source module 301 may provide thelight provided by the light source 302 toward an eye region of a userwhile changing the direction of the light at intervals of apredetermined time period under the control of the scanning mirror 303.

According to an embodiment, the light source module 301 may providelight so that the light travels from the light source 302 to the eyeregion of the user, without changing the direction of the light, underthe control of the scanning mirror 303.

The DVS camera module 304 according to an embodiment may include acamera module implemented as a DVS. The DVS camera module 304 accordingto an embodiment generates image data according to a variation in sensedlight, and does not perform data processing when there are no variations(or no variations greater than a threshold) in the sensed light.

The DVS camera module 304 according to an embodiment may generate a DVSimage, based on a variation in the amount of light reflected by the eyeregion of the user.

FIG. 20 is a block diagram of a wearable device including an eyetracking apparatus, according to an embodiment.

According to an embodiment, the glasses-type wearable device 1 may be,but is not limited to, a VR apparatus including a communication functionand a data processing function and providing a VR image or an ARapparatus including a communication function and a data processingfunction and providing an AR image.

Referring to FIG. 20, the wearable device 1 according to an embodimentmay include the eye tracking apparatus 100, the processor 120, thememory 130, a display 140, and a communication interface 180.

All of the components illustrated in FIG. 20 are not essentialcomponents of the wearable device 1. More or less components than thoseillustrated in FIG. 20 may be included in the wearable device 1.

According to an embodiment, the processor 120 may control all operationsof the wearable device 1. The processor 120 according to an embodimentmay execute one or more programs stored in the memory 130.

The memory 130 may store various pieces of data, programs, orapplications for driving and controlling the wearable device 1. Aprogram stored in the memory 130 may include at least one instruction. Aprogram (one or more instructions) or application stored in the memory130 may be executed by the processor 120.

The memory 130 according to an embodiment may store a program forprocessing and control of the processor 120, and may also store piecesof input/output data (for example, a virtual input interface, data inputvia the virtual input interface, sensing information measured by asensor, and content). The program stored in the memory 130 may beclassified into a plurality of modules according to their functions.

The memory 130 may include at least one type of storage medium among aflash memory type, a hard disk type, a multimedia card micro type, acard type memory (for example, a secure digital (SD) or extreme digital(XD) memory), a random access memory (RAM), a static random accessmemory (SRAM), a read-only memory (ROM), an electrically erasableprogrammable ROM (EEPROM), a programmable ROM (PROM), a magnetic memory,a magnetic disk, and an optical disk. The wearable device 1 may operatea web storage or a cloud server on the internet which performs a storagefunction of the memory 130. The memory 130 according to an embodimentmay store various pieces of data, programs, or applications for drivingand controlling the eye tracking apparatus 100.

The processor 120 according to an embodiment may control an operation ofthe eye tracking apparatus 100 by executing the program stored in thememory 130.

The processor 120 according to an embodiment may detect a boundary pointof the cornea of the user's eye from the DVS image generated by the DVScamera module 304. The processor 120 may filter output a peripheralregion outside the cornea region from the DVS image, based on thedetected cornea region. The processor 120 may detect a pupil region ofthe user's eye from the DVS image, based on the detected cornea region.

The processor 120 may store the detected cornea region and the detectedpupil region in the memory 130. As the positions of the detected cornearegion and the detected pupil region change, the processor 120 mayupdate the changed positions and store the changed positions in thememory 130.

As no pupil regions are detected from the DVS image, the processor 120may estimate a current position of the pupil region, based on theposition of the pupil region stored in the memory 130.

The processor 120 may determine the direction of the view of the eye,based on the positions of the cornea region and the pupil region. Theprocessor 120 may store the positions of the cornea region and the pupilregion and the determined direction of view in the memory 130.

As an operation of providing light from the light source 302 toward aneye region while changing the direction of the light under the controlof the scanning mirror 303, the processor 120 may track a change in thedirection of view of the user's eye.

The display 140 according to an embodiment may output informationprocessed by the processor 120. For example, the display 140 may displaya virtual object.

According to an embodiment, the display 140 may provide an AR image.According to an embodiment, the display 140 may include a waveguide anda display module. The waveguide may include a transparent material thatenables at least partial region of a rear surface to be visible when auser wears the wearable device 1. The waveguide may be configured as aflat plate of a single layer or multilayer structure of the transparentmaterial through which light may be reflected therein and propagated.The waveguide may face an emission surface of the display module toreceive light of a projected virtual image from the display module.Here, the transparent material includes a material through which lightmay pass, and transparency thereof may not necessarily be 100% and mayhave a certain color.

According to an embodiment, because the waveguide includes thetransparent material, the user may not only view a virtual object of thevirtual image through the display 140 but also view an external scene(e.g., real-world scene around the user). Thus, the waveguide may bereferred to as a see-through display. The display 140 may provide an ARimage by outputting the virtual object of the virtual image through thewaveguide.

The communication interface 180 may include at least one component thatenables the wearable device 1 to communicate with an external device 200of FIG. 21 or a server.

For example, the communication interface 180 may include a short-rangecommunication interface and a mobile communication interface.

Examples of the short-range communication interface may include, but arenot limited to, a Bluetooth communication interface, a short-rangewireless communication interface (e.g., a near fieldcommunication/radio-frequency identification (RFID) communicationinterface), a WLAN (i.e., WiFi) communication interface, a Zigbeecommunication interface, an infrared Data Association (IrDA)communication interface, an ultra wideband (UWB) communicationinterface, an Ant+ communication interface, and the like.

The mobile communication interface may exchange a wireless signal withat least one from a base station, an external terminal, and a server ona mobile communication network. Here, examples of the wireless signalmay include a voice call signal, a video call signal, and various typesof data according to text/multimedia messages transmission.

According to an embodiment, the wearable device 1 may transmit a DVSimage to the external device 200 of FIG. 21 via the communicationinterface 180. Accordingly, the external device 200 may detect thecornea and the pupil of the user's eye from the DVS image and maydetermine the view of the user.

According to an embodiment, the wearable device 1 may receiveinformation about positions of the cornea and pupil of the user detectedby the external device 200 of FIG. 21 and information about the viewfrom the external device 200 of FIG. 21 via the communication interface180.

FIG. 21 is a block diagram of the wearable device 1 and the externaldevice 200 according to an embodiment.

According to an embodiment, the wearable device 1 may operate inconnection with the external device 200. The wearable device 1 maytransmit the DVS image to the external device 200, and the externaldevice 200 may determine the cornea and pupil positions from the DVSimage and provide a result of eye tracking to the wearable device 1.

The components of the wearable device 1 of FIG. 21 may correspond tothose of the wearable device 1 of FIG. 20, and thus descriptions thereofwill be omitted.

The external device 200 of FIG. 21 may include a processor 220, a memory230, and a communication interface 280. However, all of the componentsillustrated in FIG. 21 are not essential components of the externaldevice 200. More or less components than those illustrated in FIG. 21may be included in the external device 200.

The processor 220 according to an embodiment may entirely control theexternal device 200. The processor 220 according to an embodiment mayexecute one or more programs stored in the memory 230.

The memory 230 according to an embodiment may store various pieces ofdata, programs, or applications for driving and controlling the externaldevice 200. A program stored in the memory 230 may include at least oneinstruction. The program (one or more instructions) or applicationstored in the memory 230 may be executed by the processor 220.

The memory 230 according to an embodiment may store a program forprocessing and control of the processor 220, and may also store piecesof input/output data (for example, a virtual input interface, data inputvia the virtual input interface, sensing information measured by asensor, and content). The programs stored in the memory 230 may beclassified into a plurality of modules according to their functions.

The communication interface 280 may include one or more components thatenable communication between the external device 200 and the eyetracking apparatus 100 or between the external device 200 and a server.

For example, the communication interface 280 may include a short-rangecommunication interface and a mobile communication interface.

Examples of the short-range communication interface may include, but arenot limited to, a Bluetooth communication interface, a short-rangewireless communication interface (e.g., a near fieldcommunication/radio-frequency identification (RFID) communicationinterface), a WLAN (i.e., WiFi) communication interface, a Zigbeecommunication interface, an infrared Data Association (IrDA)communication interface, an ultra wideband (UWB) communicationinterface, an Ant+ communication interface, and the like.

The mobile communication interface may exchange a wireless signal withat least one selected from a base station, an external terminal, and aserver on a mobile communication network. Here, examples of the wirelesssignal may include a voice call signal, a video call signal, and varioustypes of data according to text/multimedia messages transmission.

According to an embodiment, the external device 200 may receive the DVSimage from the wearable device 1 via the communication interface 280.Accordingly, the processor 220 of the external device 200 may detect thecornea region and the pupil region of the user's eye from the DVS imageand may determine the view of the user.

The external device 200 may transmit information about the determinedview of the user to the wearable device 1 via the communicationinterface 280.

The above-described embodiments may be written as computer programs andmay be implemented in general-use digital computers that execute theprograms using a computer readable recording medium. A structure of thedata used in the above-described embodiments may be recorded in acomputer readable recording medium in several ways. The above-describedembodiments may also be embodied as a storage medium includingcomputer-executable instruction codes such as computer-executableprogram modules. For example, when software modules or algorithms areinvolved, these software modules may be stored as codes or programinstructions which may be read and executed by a computer in acomputer-readable recording medium.

A computer readable medium may be any recording medium which may beaccessed by the computer and includes any volatile and/or non-volatilemedium and a removable and/or non-removable medium. Examples of thecomputer readable recording medium include, but are not limited to,magnetic storage media (e.g., ROM, floppy disks, hard disks, etc.),optical recording media (e.g., CD-ROMs, or DVDs), etc. Further, thecomputer readable medium may include all computer storage andcommunication media.

A plurality of computer readable recording media may be distributed overnetwork coupled computer systems, and data stored in the distributedrecording media, for example, program instruction words and codes, maybe executed by at least one computer.

The particular implementations shown and described herein are merelyillustrative embodiments and are not intended to otherwise limit thescope of embodiments in any way. For the sake of brevity, conventionalelectronics, control systems, software development and other functionalaspects of the systems may not be described in detail.

Although the embodiments have been disclosed for illustrative purposes,one of ordinary skill in the art will appreciate that diverse variationsand modifications are possible, without changing the technical spirit oressential features. Thus, the above embodiments should be understood notto be restrictive but to be illustrative, in all aspects. For example,respective elements described in an integrated form may be dividedlyused, and the divided elements may be used in a state of being combined.

At least one of the components, elements, modules or units describedherein may be embodied as various numbers of hardware, software and/orfirmware structures that execute respective functions described above,according to an example embodiment. For example, at least one of thesecomponents, elements or units may use a direct circuit structure, suchas a memory, a processor, a logic circuit, a look-up table, etc. thatmay execute the respective functions through controls of one or moremicroprocessors or other control apparatuses. Also, at least one ofthese components, elements or units may be specifically embodied by amodule, a program, or a part of code, which contains one or moreexecutable instructions for performing specified logic functions, andexecuted by one or more microprocessors or other control apparatuses.Also, at least one of these components, elements or units may furtherinclude or implemented by a processor such as a central processing unit(CPU) that performs the respective functions, a microprocessor, or thelike. Two or more of these components, elements or units may be combinedinto one single component, element or unit which performs all operationsor functions of the combined two or more components, elements of units.Also, at least part of functions of at least one of these components,elements or units may be performed by another of these components,element or units. Further, although a bus is not illustrated in theblock diagrams, communication between the components, elements or unitsmay be performed through the bus. Functional aspects of the aboveexample embodiments may be implemented in algorithms that execute on oneor more processors. Furthermore, the components, elements or unitsrepresented by a block or processing operations may employ any number ofrelated art techniques for electronics configuration, signal processingand/or control, data processing and the like.

What is claimed is:
 1. A wearable device comprising: a light sourcemodule comprising a light source and a scanning mirror, the light sourceconfigured to provide light toward an eye region of a user and thescanning mirror configured to control to change a direction of the lightfrom the light source toward the eye region at intervals of apredetermined time period; a dynamic vision sensor (DVS) camera moduleconfigured to sense a difference between a first amount of lightreflected by a first point of the eye region and a second amount oflight reflected by a boundary point of a cornea region of the user,which is adjacent to the first point, as the direction of the light fromthe light source to the eye region is changed over time, and generate animage representing the sensed difference between the first amount oflight and the second amount of light; and a processor configured todetect the boundary point of the cornea region of the user based on thesensed difference, represented in the image, being equal to or greaterthan a certain magnitude.
 2. The wearable device of claim 1, wherein theprocessor is further configured to detect a movement of the cornearegion by using a plurality of images consecutively generated by the DVScamera module based on variations in an amount of light reflected fromone or more points of the eye region.
 3. The wearable device of claim 2,wherein the processor is further configured to track a change in aposition of the detected cornea region based on the plurality of imagesconsecutively generated by the DVS camera module.
 4. The wearable deviceof claim 1, wherein the processor is further configured to detect apupil region of the user from the image, based on the detected cornearegion.
 5. The wearable device of claim 1, wherein the processor isfurther configured to filter a peripheral region outside the cornearegion from the image based on the detected cornea region.
 6. Thewearable device of claim 4, wherein the processor is further configuredto: store a position of the detected pupil region in a memory, and basedon a change in the position of the detected pupil region, update theposition of the detected pupil region and store the updated position inthe memory.
 7. The wearable device of claim 6, wherein the processor isfurther configured to, based on the pupil region not being detected inthe image, estimate a current position of the pupil region, based on theposition of the pupil region stored in the memory.
 8. The wearabledevice of claim 4, wherein the processor is further configured to:determine a direction of a view of an eye based on a position of thecornea region and the position of the pupil region, and store theposition of the cornea region, the position of the pupil region, and thedetermined direction of the view in a memory.
 9. The wearable device ofclaim 8, wherein the processor is further configured to track a changein the direction of the view based on a plurality of imagesconsecutively generated by the DVS camera module.
 10. A method ofoperating a wearable device, the method comprising: providing light froma light source toward an eye region of a user while changing a directionof the light at intervals of a predetermined time period by using ascanning mirror; and sensing a difference between a first amount oflight reflected by a first point of the eye region and a second amountof light reflected by a boundary point of a cornea region of the user,which is adjacent to the first point, as the direction of the light fromthe light source to the eye region is changed over time, and generate animage representing the sensed difference between the first amount oflight and the second amount of light; and detecting the boundary pointof the cornea region of the user based on the sensed difference,represented in the image, being equal to or greater than a certainmagnitude.
 11. The method of claim 10, further comprising detecting amovement of the cornea region by using a plurality of imagesconsecutively generated by the DVS camera module based on variations inan amount of light reflected from one or more points of the eye region.12. The method of claim 11, further comprising tracking a change in aposition of the detected cornea region, based on the plurality of imagesconsecutively generated by the DVS camera module.
 13. The method ofclaim 10, further comprising detecting a pupil region of the user fromthe image, based on the detected cornea region.
 14. The method of claim13, further comprising: storing a position of the detected pupil regionin a memory; and based on a change in the position of the detected pupilregion, updating the position of the detected pupil region and storingthe updated position in the memory.
 15. The method of claim 14, furthercomprising, based on the pupil region not being detected from the image,estimating a current position of the pupil region, based on the positionof the pupil region stored in the memory.
 16. The method of claim 14,further comprising: determining a direction of a view of an eye, basedon a position of the cornea region and the position of the pupil region,and storing the positions of the cornea region and the pupil region andthe determined direction of the view in the memory.
 17. The method ofclaim 10, further comprising filtering a peripheral region outside thecornea region from the image based on the detected cornea region. 18.The wearable device of claim 1, wherein the scanning mirror is furtherconfigured to change the direction of the light by controlling to changea direction in which the light is projected from the light source towardthe eye region along at least one of a longitudinal straight line, atransverse straight line, or a Lissajous curve.
 19. The method of claim10, wherein the providing the light comprises providing the light fromthe light source toward the eye region of the user while changing, byusing the scanning mirror, a direction in which the light is projectedfrom the light source toward the eye region along at least one of alongitudinal straight line, a transverse straight line, or a Lissajouscurve.